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I. 


INTRODDCTION 


A.  BACKGROUND 

This  thesis  is  an  early  step  in  a  long  term  development 
project  dedicated  to  producing  the  personal  computer  based 
Expert  System  Advisor  for  Aircraft  Maintenance  Scheduling 
(ESAAMS) .  The  development  of  expert  systems  has  rapidly 
expanded  in  the  last  decade.  These  systems  are  software 
programs  which  solve  problems  requiring  primarily  human 
expertise.  It  may  be  both  beneficial  and  cost  effective  to 
develop  and  provide  such  an  expert  system  to  organizational 
level  aviation  maintenance  managers  to  advise  and  assist 
them  with  the  challenging  task  of  aircraft  maintenance 
discrepancy  scheduling.  This  thesis  builds  directly  upon 
the  earlier  work  of  McCaffrey  [Ref.  1],  and  to  a  lesser 
extent  upon  the  works  of  Chase  [Ref.  2]  and  Allan  and 
MeSwain  [Ref.  3]. 

B.  OBJECTIVES 

The  objectives  of  this  thesis  are  to  review  the 
requirements  of  developing  an  expert  system  knowledge  base 
and  to  determine  the  suitability,  availability,  accuracy  and 
reliability  of  the  Naval  Aviation  Logistics  Data  Analysis 
(NALDA)  databases  as  the  foundation  of  unique  databases 
which  are  essential  elements  of  such  an  expert  system. 
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X.  ^  following  research  questions  will  be  addressed: 

•  What  is  an  expert  system,  how  is  one  developed,  and  how 
will  they  assist  maintenance  managers? 

•  What  is  NALDA,  where  does  it  obtain  its  data,  and  what 
type  of  information  does  it  maintain? 

•  How  timely  and  accurate  is  the  data  contained  within  the 
various  NALDA  databases? 

•  How  often  will  the  information  retrieved  from  the  NALDA 
databases  need  to  be  updated. 

•  Will  different  databases  be  required  to  support 
organizations  operating  in  different  geographical 
locations? 

•  Can  the  required  data  be  broken  out  of  the  NALDA 
databases  in  a  format  suitable  for  expert  system 
utilization? 

C.  METHODOLOGY 

Research  into  expert  systems  and  various  aspects  of  the 
NALDA  system  was  conducted  in  literature,  by  telephone 
conversation  with  Mr.  Bob  Zolio  of  the  Aviation  Supply 
Office  (ASO-045401)  and  members  of  the  NALDA  Users 
Assistance  Group,  and  by  a  personal  unstructured  interview 
with  Mr.  Gene  Woodburn  (NAMO-622C) . 

In  addition  to  this  analytical  research,  a  rudimentary 
quantitative  analysis  was  conducted  on  a  data  sample 
obtained  from  NALDA' s  Fleet  Oriented  Jobs  (FOJ)  database. 

The  data  sample  provided  various  data  elements  concerning 
approximately  78,000  individual  maintenance  actions 
performed  on  various  systems  and  sub-systems  of  Navy  and 
Marine  Corps  F/A-18  aircraft  over  a  twelve  month  period 
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ending  in  August  of  1990.  Detailed  information  concerning 
how  the  data  sample  was  developed,  definitions  of  the 
various  data  elements  are  contained  in  chapter  VI.  A  print¬ 
out  of  a  small  portion  of  the  sample  data  is  contained  in 
Appendix  A.  Copies  of  the  data  sample  diskette  and  printed 
results  of  the  two  analysis  of  variance  (ANOVA)  examinations 
performed  will  be  maintained  on  file  with  the  author  and 
Professor  Martin  J.  McCaffrey  at  the  Naval  Postgraduate 
School,  Monterey,  CA.  Data  manipulation  and  analysis  was 
conducted  on  the  Naval  Postgraduate  School  (NPS)  mainframe 
computer  using  the  Statistical  Analysis  System  (SAS) 
program . 

Analysis  of  variance  was  performed  on  mean  elapsed 
maintenance  time  (EMT)  computed  for  various  groups.  EMT  was 
chosen,  based  on  the  author's  personal  experience,  as  the 
most  important  predictor  used  in  maintenance  discrepancy 
scheduling.  An  expert  system  that  accurately  projects  the 
EMT  for  a  particular  maintenance  action  would  allow  the 
maintenance  manager  to  prioritize  the  discrepancies  to  be 
worked  upon,  to  minimize  idle  time  during  the  repair  cycle, 
and  to  maximize  utilization  of  his  activity's  limited  repair 
facilities. 
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D.  THESIS  ORGANIZATION 

The  remaining  chapters  of  the  thesis  are  as  follows: 

II.  EXPERT  SYSTEMS.  A  general  description  of  expert 
systems,  their  components,  and  their  development. 

Emphasis  is  placed  upon  the  knowledge  base  and  the  task  of 
knowledge  acquisition  as  they  pertain  to  the  development 
of  the  ESAAMS  project. 

III.  NALDA .  A  general  description  of  the  NALDA  system, 
its  various  databases,  and  the  source  of  its  data. 

IV.  DATABASE  ACCURACY.  The  accuracy  and  consistency  of 
the  NALDA  databases  is  closely  examined. 

V.  CONCLUSIONS. 
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II.  AN  INTRODUCTION  TO  EXPERT  SYSTEMS 


A.  INTRODUCTION 

This  chapter  serves  as  an  introduction  to  expert 
systems,  with  emphasis  on  the  knowledge  base  component  of  an 
expert  system  and  the  knowledge  acquisition  process  as  they 
pertain  to  the  proposed  ESAAMS  system.  The  concepts  and 
basic  elements  of  an  expert  system,  how  an  expert  system 
would  benefit  aircraft  maintenance  managers,  how  expert 
systems  differ  from  conventional  computer  programs,  the 
stages  in  the  development  of  an  expert  system,  a  variety  of 
knowledge  acquisition  techniques,  and  the  topic  of  knowledge 
validation  will  all  be  discussed. 

B.  EXPERT  SYSTEMS 

Although  the  concept  of  expert  systems  has  long  been  a 

topic  within  the  artificial  intelligence  (AI)  community,  the 

term  expert  system  has  only  recently  gained  wide  and 

accepted  use.  What  is  an  expert  system?  Dimitris  Chorafas 

defined  an  expert  system  as; 

...software  constructs  that  experts  in  specific  fields 
enrich  with  their  knowledge.  By  distilling  their 
expertise  into  sets  of  laws  and  entering  them  into 
systems,  the  experts  produce  applications  programs  that 
help  nonexperts  solve  problems  in  the  experts'  fields  by 
responding  to  program  queries.  As  such,  expert  systems 
are  artificial  intelligence  programs  that  enable  a 
computer  to  aid  you  in  a  decision-making  process.  The 
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knowhow  of  the  human  expert  is  used  to  instruct  the 
computer  how  to  solve  a  problem  or  make  a  decision. 
[Ref.  4:pp.  62-63] 

Donald  Waterman,  a  prolific  writer  on  the  svibject, 
defined  expert  systems  as: 

. . . computer  programs  that  manipulate  knowledge  to  solve 
problems  efficiently  and  effectively  in  a  narrow  problem 
area.  Like  real  human  experts,  these  systems  use 
symbolic  logic  and  heuristics-rules  of  thumb-to  find 
solutions.  And  like  real  experts,  they  make  mistakes 
but  have  the  capacity  to  learn  from  their  errors. 
However,  this  artificial  expertise  has  some  advantages 
over  human  expertise:  It  is  permanent,  consistent,  easy 
to  transfer  and  document,  and  cheaper.  In  sum,  by 
linking  the  power  of  computers  to  the  richness  of  human 
experience,  expert  systems  enhance  the  value  of  expert 
knowledge  by  making  it  readily  and  widely  accessible. 
[Ref.  5:p.  XVII] 

Finally,  M.  A.  Bramer  defined  an  expert  system  simply 


as: 


...a  computing  system  which  embodies  organized  knowledge 
concerning  some  specific  area  of  human  expertise, 
sufficient  to  perform  as  a  cost-effective  consultant. 

[Ref.  6:p.  3] 

Bramer  expanded  on  his  definition  by  stating  that  expert 
systems  usually  make  use  of  heuristic  rules,  relating  to  the 
specialty  in  question,  which  are  obtained  from  subject 
matter  experts  and  refined  through  experience. 

Heuristic  is  a  term  derived  from  the  Greek  word 
"heuriskein” ,  meaning  to  discover.  A  heuristic  is  a  rule  of 
thumb,  a  technique  or  a  simplification  that  limits  or 
narrows  the  search  procedure. 

The  earliest  acknowledged  expert  system,  DRENDAL,  was 
first  developed  in  the  1965  [Ref.  6:p.  3].  Two  other  early 
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and  comparatively  well  known  expert  systems,  both  developed 
at  the  Stanford  Research  Institute,  are  MYCIN  (a  medical 
diagnostic  tool) ,  and  PROSPECTOR  (a  geological  exploration 
tool) .  Another  well  known,  and  more  recently  developed, 
expert  system  is  ACE  (Automated  Cable  Expertise) ,  developed 
by  AT&T  and  in  use  since  1982.  Currently  expert  systems  are 
widely  utilized  in  many  fields  and  are  gaining  popularity  in 
the  military,  manufacturing,  banking  and  service  industries. 
[Ref.  7:p.  69] 

Despite  their  growing  use  and  the  positive  nature  of  the 
definitions  above,  readers  should  be  aware  that  expert 
systems  are  not  the  answer  to  every  problem  and  do  have 
their  drawbacks.  Expert  systems  are  designed  to  perform 
tasks  that  require  cognitive  as  opposed  to  physical  skills. 
If  the  task  can  only  be  learned  through  practicing  physical 
manipulations  then  it  is  unlikely  that  the  expert  system 
approach  will  succeed.  [Ref.  5:p.  128]  The  development  of 
an  expert  system  is  an  extremely  complex  and  time  consuming 
operation.  Extracting  expert  knowledge  is  a  challenging 
process.  Finally,  even  with  the  best  experts  contributing 
to  their  development,  knowledge  based  systems  are  not  100% 
reliable.  [Ref.  7:p.  74] 
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C.  DIFFERENCES  BETWEEN  EXPERT  SYSTEMS  AND  CONVENTIONAL 

PROGRAMS 

Valerie  Barr  of  the  Pratt  Institute  writes  that: 

In  expert  systems,  unlike  other  types  of  programs,  we 
tell  the  computer  what  to  know,  not  what  to  do.  When 
constructing  the  expert  system,  we  do  not  provide  a  set 
of  instructions;  rather  ,  we  provide  knowledge  and 
advice.  If  we  have  a  step-by-step  algorithm  for  solving 
the  problem  at  hand,  then  a  "traditional"  program  is  in 
order.  However,  if  we  have  no  such  step-by-step  method, 
then  an  expert  system  is  in  order.  [Ref.  8:p.68] 

Some  readers  may  confuse  an  expert  system  with  the  more 
widely  known  decision  support  systems  (DSS)  or  database 
management  systems  (DBMS) .  The  key  difference  is  that  as 
DSSs  have  been  constructed  on  the  knowledge  acquired  through 
DBMS  applications,  expert  systems  are  built  upon  the 
foundation  of  knowledge  gained  through  the  development  and 
implementation  of  DSSs.  [Ref.  4:p.  67]  An  easy  to  remember 
and  rudimentary  difference  between  expert  systems  and  DSS  or 
DBMS  programs  is  that  while  the  conventional  programs 
manipulate  data,  expert  systems  manipulate  knowledge  [Ref. 
7:p.  67]. 

D.  BENEFITS  OF  DEVELOPING  ESAAMS 

McCaffrey  determined  that  the  aircraft  maintenance 
discrepancy  scheduling  domain  was  acceptable  for  an  expert 
system  solution  and  that  developing  an  expert  system  to 
assist  aircraft  maintenance  managers  would  be  beneficial  to 
the  U.S.  Navy  [Ref.  l:pp.  107-111].  The  development  of  a 
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knowledge  base  of  shared  expertise  would  partially  solve  the 
learning  curve  problem  many  managers  experience  when 
assigned  to  a  squadron  possessing  aircraft  they  are  not 
familiar  with. 

Additionally,  expert  system  development  offers  one  of 
the  few  methods  available  with  the  potential  to  achieve 
significant  gains  in  operational  readiness.  Because  of  the 
large  aircraft  inventory  possessed  by  the  Navy  and  Marine 
Corps,  a  gain  of  as  little  as  one  percent  would  translate 
into  an  additional  50  operationally  ready  aircraft  per  day. 
[Ref.  1,  pp.  110-111] 

E.  COMPONENTS  OF  AN  EXPERT  SYSTEM 

An  expert  system  consists  of  three  basic  components 
[Ref.  7:pp.  76-77]; 

•  A  knowledge  bank  (sometimes  referred  to  as  an 
information  base  or  a  knowledge  base) ,  containing  task 
specific  facts  and  rules.  In  a  rule  based  expert 
system,  the  knowledge  base  breaks  down  into  the  rule 
base  and  the  working  memory  (or  data  base) . 

•  An  inference  engine,  containing  problem  solving  methods 
and  a  mechanism  for  using  the  knowledge  base's 
information. 

•  A  user  interface,  which  allows  the  user,  the  maintenance 
manager  in  the  case  of  ESAAMS,  to  interact  with  the 
program . 

Frequently,  rather  than  duplicate  data  already  stored  in 
existing  databases  within  the  knowledge  base,  an  expert 
system  will  be  designed  to  import  data  from  a  database  when 
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needed.  A  diagram  illustrating  the  major  components  of  the 
ESAAMS  system,  including  the  NALDA  databases,  is  provided  in 
Figure  1. 


MAINTENANCE 

MANAGER 


Figure  1  ESAAMS  System  Components 

The  two  primary  components  of  any  expert  system,  which 
require  the  majority  of  the  development  effort,  are  the 
inference  engine  and  the  knowledge  base.  The  function  of 
the  inference  engine  is  hypothesis  proving.  The  inference 
engine  is  software  that  implements  a  search  of  the  rules 
contained  in  the  knowledge  base,  looking  for  matches  to  the 
initial  information  given  in  the  working  memory.  The 
inference  engine  controls  the  process  of  invoking  rules. 
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deciding  how  to  apply  the  rules  to  infer  new  knowledge  and 
in  which  order  the  rules  should  be  applied. 

Developers  of  expert  systems  today  can  choose  from  a 
number  of  commercially  available  expert  system  shells.  An 
expert  system  shell  is  a  program  containing  a  working, 
tested  and  debugged  inference  engine  and  user  interface  in  a 
framework  into  which  the  developer  can  program  the  unique 
knowledge  for  his  particular  application  into  the  knowledge 
base.  [Ref.  8:pp.  68-69]  The  obvious  advantage  of  using  an 
expert  system  shell  is  that  it  allows  the  developer  to 
concentrate  on  acquiring  and  refining  the  knowledge  base 
instead  of  the  computer  code  necessary  to  produce  a 
functioning  inference  engine. 

The  knowledge  base,  sometimes  referred  to  as  the 

knowledge  bank  in  order  to  avoid  confusion  with  the  term 

"database",  is  the  heart  of  any  expert  system.  Walters  and 

Nielsen  defined  a  knowledge  base  as  containing: 

...all  the  application-specific  information.  This 
information  can  be  in  the  form  of  simple  facts  (e.g., 
data  names  and  values),  relationships  (e.g.,  parent- 
child  class  memberships) ,  or  procedural  information 
(e.g. ,  sequential  code  for  printing  a  report  or  drawing 
a  graph).  [Ref.  9:p.  5] 

The  knowledge  base  contains  facts  in  its  working  memory, 
and  rules  in  its  rule  base.  Rules  can  be  thought  of  as  long 
term  information  that  rarely,  if  ever,  changes.  Conversely, 
facts  are  short  term  information  that  can  change  quite 
often.  Today,  most  commercial  and  experimental  expert 
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systems  use  the  IF-THEN  rule  format.  Rules  are  formatted 
into  two  separate  parts.  The  first  part  of  the  rule  (IF) 
states  some  premise  or  condition.  The  IF  part  of  the  rule 
may  contain  more  than  a  single  premise  and,  if  so,  will 
contain  a  clause  for  each.  These  are  called  compound 
clauses  and  are  linked  by  conjunctions  AND  or  OR.  The 
second  part  of  the  rule  (THEN)  states  a  conclusion, 
consequence,  or  action  that  will  occur  if  the  conditions  of 
the  first  part  of  the  rule  have  been  met.  [Ref.  7:pp.  77-78] 
For  example: 

IF  the  animal  lives  in  water 

AND  the  animal  breathes  water 

THEN  the  animal  is  a  fish,  CF  1.0 

The  CF  stands  for  certainty  factor,  and  is  sometimes 
referred  to  as  a  confidence  factor  or  rule  strength.  It  is 
a  number  between  0  and  1  representing  the  confidence  we  have 
in  the  validity  of  our  conclusion.  A  certainty  factor  is 
not  a  probability,  it  is  simply  a  number  that  represents  the 
degree  of  uncertainty  you  have  in  a  particular  rule. 
Certainty  factors  are  determined  by  the  human  experts 
creating  the  knowledge  base.  The  certainty  factors  can  be 
based  upon  either  intelligent  estimates,  statistical  data, 
or  a  combination  of  the  two.  In  rules  with  compound  premise 
clauses  connected  by  either  AND  or  OR,  each  clause  may  have 
its  own  certainty  factor  and  the  programmer  must  determine  a 
means  to  compute  a  composite  certainty  factor  for  the  rule. 
[Ref.  7:pp.  86-88] 
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F.  EXPERT  SYSTEM  DEVELOPMENT 


There  are  several  methods  for  developing  expert  systems. 
This  report  will  summarize  a  scheme  published  by  Juan  J. 
Ferrada  and  John  M.  Holmes.  [Ref.  10:pp.  35-41]  The 
development  of  an  expert  system  can  be  simplified  to  four 
global  steps: 

•  Defining  the  problem 

•  Preliminary  prototyping 

•  Developing  the  expanded  prototype 

•  Delivering  the  system 

1.  Defining  the  Problem 

The  objectives  of  this  step  are  to  ensure  that:  the 
problem  to  be  solved  is  clearly  defined;  the  project  will 
satisfy  a  real  need;  the  project  is  technically  feasible; 
the  functions  to  be  carried  out  are  specified;  the 
knowledge  base  requirements  are  established;  and  the 
available  expert  system  development  tools  are  analyzed. 

After  the  above  questions  have  been  answered,  a  decision 
must  be  made  whether  to  use  an  expert  system  shell  or  an  AI 
computer  language.  Expert  systems  can  be  constructed 
utilizing  any  number  of  commercially  available  expert  system 
shells  or  AI  computer  languages.  An  expert  shell  has  the 
advantage  of  containing  a  substantial  amount  of  AI  code  and 
an  inference  engine  that  has  been  tested,  debugged  and 
maintained.  Building  an  expert  system  using  an  AI  computer 


13 


language  provides  the  expert  system  developer  wich  greater 
flexibility  while  simultaneously  tasking  him  with  writing  a 
considerable  amount  of  code. 

The  final  requirement  of  this  initial  step  is  to  define 
the  available  sources  of  expert  knowledge.  A  knowledge 
engineer  is  the  person  responsible  for  gathering  and 
organizing  the  expert  knowledge  into  an  expert  system 
knowledge  base.  The  knowledge  engineer  must  identify  the 
knowledge  base  rec^airements  as  well  as  the  reliability  and 
availability  of  human  experts,  databases,  spreadsheets, 
descriptions  in  text  files,  graphics  and  external  computer 
programs.  The  topic  of  knowledge  acquisition  will  be 
discussed  in  greater  detail  later  in  this  thesis. 

2.  Preliminary  Prototyping 

The  principal  objective  of  this  step  is  to  quickly 
demonstrate  the  technical  and  economic  feasibility  of  the 
expert  system  under  development.  Preliminary  prototype 
testing  should  be  analyzed  to  determine  how  well  the 
interface  relates  to  the  knowledge  base,  how  well  the 
inference  engine  can  adapt  to  the  way  the  knowledge  has  been 
introduced  to  the  expert  system,  how  well  the  knowledge  base 
has  been  organized  by  the  knowledge  engineer,  and  how  well 
the  proposed  environment  for  delivery  (PC,  minicomputer  or 
mainframe)  performs.  As  recently  as  1987  most  writers 
concluded  that  useful  expert  systems  had  to  implemented  and 
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delivered  on  minicomputers  or  mainframes  [Ref.  7:p.  74]. 
Currently  personal  computers,  which  are  quickly  migrating  to 
the  386,  the  486  and  even  more  powerful  chips,  have  begun  to 
represent  the  single  largest  segment  of  the  expert  system 
business  and  are  the  key  to  the  commercial  acceptance  of 
expert  system  technology  [Ref.  ll:p.  56]. 

3.  Developing  the  Expanded  Prototype 

The  objective  of  this  step  is  to  develop  a  system 
capable  of  convincing  management  that  it  is  a  useful  and 
potentially  valuable  tool.  The  expanded  prototype  should 
include  an  expansion  of  the  knowledge  base,  sophistication 
of  representation,  and  links  to  other  systems  and  programs. 
The  system  should  be  capable  of  dealing  with  a  variety  of 
exceptions  and  special  cases  ignored  during  earlier 
prototyping.  If  the  expanded  prototype  performs  well  the 
development  of  a  more  extensive  delivery  system  may  not  be 
required . 

4.  Delivering  the  System 

During  this  phase  the  developer  analyzes  the  different 
delivery  environments  mentioned  above  and  optimizes  the 
program  requirements  in  terms  of  memory  usage  or 
performance. 

G.  KNOWLEDGE  ACQUISITION  AND  REFINEMENT 

Expert  systems  are  powerful  and  valuable  instruments 
because  of  the  capacity  they  gain  from  the  knowledge  they 


15 


incorporate.  The  development  of  a  reliable  and  accurate 

knowledge  base  is  the  primary  goal  of  any  knowledge  engineer 

tasked  with  developing  an  expert  system.  Knowledge 

acquisition  and  refinement  have  historically  accounted  for  a 

major  portion  of  the  overall  development  expense  and  time  of 

most  expert  svstems.  Knowledge  acquisition  is  defined  as; 

...the  transfer  of  expertise  from  a  human  expert  to  a 
machine.  When  the  machine,  extends  its  initial 
knowledge  by  learning  methods,  we  refer  to  this  as 
knowledge  refinement  [Ref.  12:p.  320]. 

As  the  demand  for  expert  systems  has  multiplied,  the 

limited  availability  of  qualified  and  skilled  knowledge 

engineers  has  made  knowledge  acq’aisition  appear  to  be  a 

major  bottleneck  in  the  construction  of  new  expert  systems. 

However,  this  assumption  is  based  on  the  dated  view  that 

knowledge  engineering  is  a  manual  craft,  depending  on  the 

skill  of  a  knowledge  engineer  who  is  armed  with  only  a  pen, 

paper  and  tape  recorder.  The  key  to  solving  this  knowledge 

acquisition  bottleneck  is  to  automate  some  portion  of  the 

knowledge  engineering  tasks.  [Ref.  13 :p.  49] 

Traditionally,  knowledge  acquisition  has  been  performed 

by  a  knowledge  engineer  who  interviews  an  expert,  identifies 

the  components  of  his  knowledge,  and  builds  the  expert 

system  knowledge  base.  The  knowledge  engineer's  task  is 

complicated  by  the  fact  that  human  experts  have  rarely 

analyzed  their  thoughts  and  the  structure  of  their 

knowledge,  and  can  rarely  provide  an  overall  account  of  how 
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a  decision  is  made.  Traditionally  knowledge  obtained  from 
any  source  other  than  a  human  expert  was  discounted  when 
developing  an  expert  system.  Frenzel  stated,  in  1987,  that: 

.knowledge  comes  in  many  forms.  It  can  be  standard 
textbook  knowledge  that  you  can  dig  out  of  books, 
articles,  and  other  references  quickly  and  easily.  This 
knowledge  is  important,  but  is  usually  not  the  best  kind 
of  knowledge  for  an  expert  system.  The  real  Icnowledge 
will  come  from  individuals  who  are  experts  in  the 
subject.  [Ref.  7:p.  105] 

The  traditional  interview  approach  to  knowledge 
acquisition  has  several  shortcomings.  One  is  the  afore 
mentioned  shortage  of  skilled  knowledge  engineers.  Another 
is  the  growth  in  size  of  expert  systems  knowledge  bases. 
Early  expert  systems  often  used  less  than  100  rules,  and  the 
use  of'more  than  300  rules  was  rare  [Ref.  14:p.  44]. 

Current  state-of-the-art  expert  systems  may  contain  tens  of 
thousands  of  rules  in  their  knowledge  bases  [Ref.  15:p.  15]. 
McCaffrey  estimated  the  knowledge  base  of  the  proposed 
expert  system  would  contain  approximately  1000  to  2000  rules 
[Ref.l:p.  122].  Clearly  automating  some  or  all  of  the 
knowledge  acquisition  process  is  called  for  in  order  to 
overcome  the  "knowledge  engineering  bottleneck." 

Parsaye,  who  defines  knowledge  acquisition  as  "the 
transfer  of  problem-solving  expertise  from  some  knowledge 
source  to  a  program,"  identifies  three  basic  approaches  to 
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knowledge  acquisition  currently  available  [Ref.  13:p.50]: 

•  interviewing 

•  learning  by  interaction 

•  learning  by  induction 

1 .  Interviewing 

This  technique  involves  the  knowledge  engineer  acquiring 
knowledge  from  a  human  expert  through  a  sequence  of 
interviews  and  then  encoding  the  acquired  knowledge  in  the 
expert  system's  knowledge  base.  The  knowledge  engineer 
plays  a  central  role  in  this  process  and  his  skills  largely 
determine  the  (juality  of  the  information  obtained  and 
entered  into  the  knowledge  base. 

Most  readers  will  recognize  that  while  the  interview 
process  is  an  easy-to-apply  and  flexible  approach,  it  also 
contains  some  serious  drawbacks:  interviewing  is  a  time 
consuming  and  subjective  process;  often  the  knowledge 
engineer,  with  only  a  limited  amount  of  subject  matter 
expertise,  will  not  explicitly  understand  important 
concepts,  resulting  in  an  incomplete  or  inaccurate  knowledge 
base  structure;  and  directly  asking  questions  of  experts 
risks  altering  their  view  of  what  they  actually  do.  with 
few,  if  any,  knowledge  engineers  familiar  with  the  complex 
task  of  aircraft  maintenance  scheduling,  these  drawbacks 
will  be  particularly  applicable  to  the  development  of  the 
ESAAMS  project. 


2.  Learning  by  Interaction 

This  knowledge  acquisition  technique  relies  on  computer 
assistance.  In  essence  the  human  expert  interacts  directly 
with  a  computer  program  that  captures  his  expert  knowledge. 
This  process  focuses  on  specific  interactive  methodologies 
and  interviewing  techniques  that  help  experts  clarify  the 
structure  of  their  own  thoughts  and  knowledge.  The  need  for 
a  knowledge  engineer  can  be  significantly  reduced  by 
utilizing  the  learning  by  interaction  technique. 

3.  Learning  by  Induction 

In  this  knowledge  acquisition  process  a  computer  program 
distills  knowledge  by  examining  data  and  examples.  The 
basic  concept  is  to  have  the  program  learn  to  perform  a  task 
by  analyzing  the  data  documenting  the  human  expert’s 
performance.  The  primary  difficulty  with  learning  by 
induction  is  identifying  suitable  attributes  and 
characteristics  on  which  to  perform  the  induction.  The 
primary  advantages  of  this  technique  are  that  dependence  on 
knowledge  engineers  and  human  experts  is  diminished,  and 
knowledge  bases  containing  large  numbers  of  rules  can  be 
developed  in  a  short  period  of  time. 

4 .  Summary 

All  three  knowledge  acquisition  techniques  have  their 
advantages  and  disadvantages.  The  majority  of  expert 
systems  currently  under  development  will  use  all  three 
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techniques  in  the  development  of  their  knowledge  bases.  The 
one  factor  all  three  techniques  have  in  common,  and  an 
important  source  of  expert  knowledge,  is  historical  data 
about  how  an  expert  actually  performed  a  task.  Much  of  this 
data  is  contained  in  computerized  databases,  such  as  the 
various  NALDA  databases.  A  fundamental  difficulty  with 
using  any  of  these  databases  is  the  obstacle  of  connecting 
the  AI  computer  language  or  the  expert  shell  to  various 
external  programs  and  databases.  William  Martorelli  expands 
on  this  challenge: 

For  expert  systems  to  be  most  useful,  they  should  be 
able  to  communicate  with  existing  systems  and  data 
sources.  Today's  PC-based  (expert  system)  applications 
are  still  often  stand-alone  entities,  or  they  possess 
rudimentary  links  to  spreadsheets  or  PC  databases.  [Ref. 
ll:p.  62] 

A  number  of  vendors  are  striving  to  make  their  expert 
system  development  tools  more  readily  connectable  to 
external  programs  and  data  bases.  Ken  Pedersen  synopsized 
the  connection  capabilities  of  11  commercially  available  PC- 
based  expert  system  shells  in  Reference  16.  Readers  who 
desire  a  more  thorough  documentation  of  the  connection 
capabilities  of  these  shells  are  referred  to  The  PC  Expert 
Systems  Shoot-Out,  published  by  Expertise  Associates  of  W. 
Lafayette,  Indiana.  In  general,  many  of  the  shells 
available  on  the  market  today  can  connect  to  such  popular  PC 
based  programs  as  Lotus  1-2-3  and  dBase.  Many  of  the  expert 
system  shells,  such  as  VP-Expert,  can  offer  an  induction 
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module  capable  of  translating  record  based  examples  into 
production  rules.  The  resulting  knowledge  base  is  only  as 
good  as  the  data  file  on  which  it  is  based. 


H.  KNOWLEDGE  VALIDATION 

A  fundamental  question  concerning  every  expert  system  is 
how  applicable  is  the  expert  knowledge  acquired  for  your 
system?  Knowledge  base  validation  is  an  area  of  great 
concern  to  all  knowledge  engineers.  Validation  of  the 
knowledge  base  should  be  viewed  as  an  ongoing  process  built 
into  the  development  effort.  The  two  primary  goals  of  any 
software  validation  effort  are  to  ensure  that:  (1)  the 
program  performs  the  functions  intended,  and  (2)  the  program 
does  hot  perform  any  intended  function  that  could  adversely 
affect  the  performance  of  the  entire  system.  [Ref.  17 :p. 

287]  Parsaye  expands  on  this  topic: 

Intuitively  the  aim  of  any  validation  effort  is  to 
ensure  that  the  expert  system  behaves  correctly,  but 
what  does  this  mean  for  systems  that  produce  inexact 
results?  Can  we  ever  be  100%  sure  the  knowledge  is 
correct? 

From  a  philosophical  viewpoint  the  answer  is  no.  Even 
in  traditional  programs,  you  can  never  be  100%  sure 
about  a  verification  effort  in  all  cases  since  as  the 
program  size  grows  the  size  of  the  verification  program 
will  also  grow,  and  at  some  point  the  verification 
system  may  be  more  complex  and  error  prone  than  the 
program  itself.  [Ref.  13 :p.  59] 

Parsaye  further  states  that  the  two  key  questions  to  ask 
during  validation  are:  (1)  How  often  are  mistakes  made?, 
and  (2)  What  is  the  price  to  be  paid  for  each  mistake? 
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Parsaye's  approach  is  to  compare  the  performance  of  the 
expert  system  under  development  against  that  of  a  h\iman 
expert,  or  even  another  expert  system,  on  a  series  of  test 
cases.  Because  the  overall  measure  of  performance  obtained 
under  these  circumstances  would  depend  largely  on  the 
quality  of  the  test  cases,  it  is  important  to  also  measure 
the  quality  of  the  test  cases.  Utilizing  a  more 
representative  set  of  test  cases  will  furnish  the  developer 
with  a  more  significant  set  of  results. 


I.  SUMMARY 

This  chapter  has  provided  the  reader  with  a  large  amount 
of  information  concerning  expert  systems  and  their 
development.  The  three  most  significant  points  to  take  from 
this  chapter  are  that: 

•  With  the  ready  availability  of  commercially  proven 
shells,  the  most  challenging  task  facing  today's  expert 
system  developer  is  to  acquire,  develop  and  validate  the 
applicable  knowledge  base  rules. 

•  Whichever  combination  of  knowledge  acquisition 
technic[ues  is  used,  a  data  base  of  information 
concerning  past  expert  performance  is  required.  The 
quality  of  the  knowledge  base  developed  will  depend 
largely  on  the  accuracy,  timeliness  and  completeness  of 
the  information  contained  in  the  database. 

•  Connectability  with  the  NALDA  system  should  be 
considered  when  choosing  an  expert  system  shell  for  use 
on  the  ESAAMS  project.  Because  NALDA  has  the  capability 
to  download  data  via  floppy  diskette  in  ASCII  format, 
indirect  connection  through  widely  available  software, 
such  as  dBase  or  Lotus  1-2-3,  with  many  currently 
available  expert  system  shells  is  likely.  Further 
research  into  the  direct  connection  of  the  NALDA  system 
with  an  expert  system  shell  is  warranted. 


III.  NALDA 


A.  INTRODUCTION 

This  chapter  describes  the  NALDA  system,  including  the 
source  of  NALDA 's  data  and  the  various  databases  it 
maintains.  The  following  specific  questions  will  be 
addressed:  What  is  NALDA?  Where  does  NALDA  obtain  its 

data?  What  type  of  information  is  stored  in  the  various 
NALDA  databases?  Which  database  is  the  most  likely  source 
of  information  for  the  unique  databases  which  will  become 
essential  elements  of  the  ESAAMS  system. 

B.  NALDA  DEFINED 

The  NALDA  system,  is: 

...an  automated  database  and  information  retrieval 
system  for  aviation  logistics  management  and  technical 
decision  support.  Analysis  capability  is  provided 
through  interactive  query  and  batch  processing  from 
remote  terminals.  [Ref.  18 :p.  I-l] 

The  principal  objective  of  the  NALDA  system  is  to  supply 
a  state-of-the-art  management  information  system  to  assist 
Naval  Aviation  maintenance  and  logistics  managers  in  making 
improved  decisions  affecting  U.S.  Naval  Aviation  readiness. 
To  accomplish  this  objective  the  NALDA  system  provides  a 
centralized  data  bank,  including  remote  terminals  with 
maintenance  data  retrieval  and  analysis  capabilities,  that 
can  solve  complex  integrated  logistics  support  problems  for 
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which  other  available  means  have  consistently  proved 
inadequate.  [Ref.  18 ;p.  I-l]  Overall  management  of  the 
NALDA  program  is  conducted  by  the  Naval  Air  Systems  Command, 
while  day-to-day  operations  are  coordinated  by  the  Naval 
Aviation  Maintenance  Office  (NAMO) ,  located  aboard  Naval  Air 
Station  Patuxent  River,  Maryland. 


C.  NALDA  DATA  SOURCES 

NALDA  incorporates  data  from  an  assortment  of  different 
sources  including  Naval  Aviation  Depots,  the  Aviation  Supply 
Office,  and  the  Naval  Safety  Center  [Ref.  18;p.  I-l].  The 
principal  source  for  NALDA  data  is  the  monthly  Aviation 
Maintenance  and  Material  Management  (3-M)  summary  produced 
by  the  Naval  Aviation  Maintenance  Support  Office  (NAMSO) . 

The  source  for  the  3-M  summary  is  the  Navy's  Maintenance 
Data  System  (MDS) .  The  Naval  Aviation  Maintenance  Plan 
(NAMP) ,  separates  the  MDS  data  flow  into  three  cycles  [Ref. 
19: para.  2.1.3  -  2.1.7]: 

•  Local  Cycle:  During  this  cycle  data  elements  concerning 
maintenance  actions  performed  are  entered  onto  source 
documents  by  technicians  at  either  the  organizational  or 
intermediate  level. 

•  Local/central  Cycle:  During  this  cycle  the  completed 
source  documents  are  screened,  corrected,  and  converted 
into  machine  language  by  a  supporting  data  services 
facility  (DSF) . 

•  Central /external  Cycle:  During  this  cycle  data  from  the 
numerous  DSFs  are  collected  and  processed  by  NAMSO,  and 
various  reports  are  issued  to  originating  activities  and 
the  chain  of  command. 
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1.  Local  Cycle 

The  local  cycle  of  the  MDS  is  designed  so  that  each 
technician,  when  performing  a  task,  converts  a  narrative 
svunmary  of  the  task  into  a  series  of  alphanvuneric  and 
numeric  codes  and  enters  the  coded  information  on  to  one  of 
two  source  documents,  either  a  Support  Action  Form  (SAF)  or 
a  Visual  Information  Display  System/Maintenance  Action  Form 
(VIDS/MAF) .  Source  documents  are  screened  for  accuracy  and 
completeness  by  the  technician's  work  center  supervisor, 
maintenance  control  supervisor,  and  the  activity’s  data 
analyst.  These  source  documents  are  then  collected  and 
transported  to  a  local  data  services  facility  (DSF) ,  located 
aboard  either  the  host  air  station  or  deployed  ship,  where 
the  information  is  converted  to  machine  record.  The  DSFs 
screen  the  incoming  source  documents  and  any  questionable 
data  are  referred  back  to  the  originating  activity  for 
verification.  The  DSF  uses  the  machine  records  to  produce  a 
number  of  periodic  reports  summarizing  the  submitted  data. 
These  reports  are  used  to  verify  the  data,  for  local  unit 
analysis,  and  for  maintenance  planning. 

2 .  Local/Central  Cycle 

After  verification,  duplicate  record  files  of  the 
information  are  generated  and  transmitted  to  NAMSO,  Naval 
Aviation's  central  collection  facility.  At  NAMSO  the  data 
received  is  combined  with  data  received  from  other  DSFs. 
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Machine  runs  for  errors  which  are  detectable  by  computer  are 
made,  and  the  results  of  this  error  analysis  are  forwarded 
back  to  the  DSFs  and  originating  activities. 

3 .  Central/Extemal  Cycle 

In  addition  to  providing  periodic  3-M  feedback  reports 
to  the  originating  activities,  NAHSO  provides  data  and 
reports  to  various  agencies,  including  the  Chief  of  Naval 
Operations  (CNO) ,  various  systems  command  field  agencies, 
NALDA,  and  contractors. 

D.  NALDA  DATABASES 

NALDA  maintains  a  number  of  specialized  processing 
systems  and  databases.  These  systems  and  databases  are 
designed  to  provide  system  users  with  the  necessary 
information  to  solve  problems  and  to  make  informed 
management  decisions.  [Ref*  18:pp.  1-2  to  1-3]  These 
databases  include : 

•  Fleet  Oriented  Jobs  fFOJ) ;  contains  the  most  recent 
eighteen  months  of  selected  maintenance,  material, 
readiness,  inventory  and  operations  data  and  are  the 
principal  repository  of  source  data  in  the  NALDA  system. 

•  Ecruipment  Condition  Analysis  fECA) ;  Provides  users  with 
3-M  data  and  flight  data,  as  far  back  as  1974. 

•  Technical  Directive  Status  Accounting  fTDSA^ ;  Stores, 
maintains,  and  disseminates  information  concerning  the 
incorporation/non-incorporation  of  Technical  Directives. 
Provides  projected  and  actual  man-hour  reporting, 
configuration  status  of  equipment  items,  and  change-kits 
material  accounting. 

•  Support  Action  Form  fSAF) ;  Contains  the  most  recent 
eighteen  months  of  support  action  data. 
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•  Depot  oriented  Jobs  ^DOJ) ;  Contains  data  collected  by 
the  Depot  Maintenance  Data  System  related  to  jobs 
originated  by  the  depots. 

•  Intermediate  Maintenance  Activity  Analysis  flMA^ ; 
Contains  summary  data  that  facilitates  analysis  of 
production  data  at  the  intermediate  level  of  maintenance 
and  at  individual  IMAs  for  items  identified  by  either  a 
National  Stock  Nvunber  or  a  part  number. 

•  Utilization  fUTIL^ :  Contains  the  most  recent  eighteen 
months  of  utilization  and  aircraft  flight/flight  hour 
data. 

•  NALDA/Aircraft  Engine  Management  System  fNALDA/AEMS) : 
Contains  information  derived  from  the  Navy's  on-line 
AEMS. 

•  Equipment  Summary  Reports  (ESR^ :  Contains  current 
summary  information  which  is  identical  to  the 
information  contained  in  Aviation  3-M  reports. 

•  Scheduled  Removal  Component  fNEWSRC^ ;  Contains  data 
from  Scheduled  Removal  Component  (SRC) ,  Assembly  Service 
Record,  Module  Service  Record  and  History  Record  forms 
which  are  collected  at  the  SRC  Central  Repository. 


E.  THE  MOST  LIKELY  SOURCE  OF  DATA  FOR  ESAANS 

After  interviewing  ntimerous  personnel  working  in  the 
NALDA  User  Assistance  Group,  this  author  determined  that  the 
most  probable  source  for  the  majority  of  the  data  needed  for 
the  ESAAMS  system  knowledge  base  was  the  FOJ  database. 

Based  on  this  determination,  further  research  and  analysis 
concentrated  on  the  FOJ  database.  Other  specialized 
databases  maintained  by  NALDA  could  be  utilized,  depending 
upon  the  information  required  and  the  scope  of  the  proposed 
ESAAMS  system. 
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F.  SUMMARY 


This  chapter  has  presented  the  reader  with  a  large 
amount  of  information  concerning  the  NALDA  system  and  the 
databases  it  maintains.  The  key  point  to  take  from  this 
chapter  is  that  the  NALDA  databases  are  Naval  Aviation's 
central  repository  of  logistical  and  maintenance 
information.  Because  they  gather  historical  information 
from  a  wide  variety  of  sources  throughout  the  fleet,  and 
because  they  are  uniguely  organized  to  handle  ad  hoc 
queries,  they  are  the  most  likely  source  of  data  for  the 
unique  databases  which  will  become  essential  elements  of  the 
ESAAMS  system. 
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IV.  SUITABILITY  AND  ACCDSACY 


A.  INTRODUCTION 

This  chapter  evaluates  the  suitability  and  accuracy  of 
the  NALDA  databases  as  the  foundation  for  the  knowledge  base 
of  the  ESAAMS  system.  A  description  of  the  data  sample  used 
in  this  evaluation  process  is  provided.  The  following 
specific  research  questions  are  answered:  Can  the  data  be 
broken  out  in  a  useful  form?  How  dated  is  the  NALDA  data? 
How  accurate  is  the  data?  How  does  geographical  location  of 
the  originating  activities  affect  the  data?  How  often  would 
a  knowledge  base  founded  upon  the  NALDA  databases  need  to  be 
updated? 

B.  DATA  SAMPLE 

A  data  sample,  comprising  records  of  approximately 
78,000  individual  F/A-18  maintenance  actions,  was  obtained 
from  the  NALDA  Users  Assistance  Group.  A  print-out  of  a 
portion  of  this  data  sample  is  attached  as  Appendix  A.  The 
following  information  was  contained  in  each  record:  Job 
Control  Number  (JCN) ,  Work  Unit  Code  (WUC) ,  Transaction  Code 
(TRANS) ,  Malfunction  Code  (MAL) ,  and  Elapsed  Maintenance 
Time  (EMT) .  Another  data  element.  Maintenance  Man-Hours 
(MMH) ,  was  provided  in  the  data  sample  but  was  not  used  in 
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the  analysis.  A  brief  description  of  each  data  element 
follows. 


1.  Job  Control  Number  (JCN) 

The  JCN  is  a  9-,  10- ,  or  11-character  alphanumeric  code 
that  serves  as  a  base  for  MDR  and  Maintenance  control 
procedures.  The  JCN  allows  for  separate  identification  of 
each  maintenance  action,  and  provides  a  link  with  the 
maintenance  actions  performed  by  the  IMA  (Intermediate 
Maintenance  Activity)  in  support  of  an  activity  or  an  O- 
level  (Organizational  level)  maintenance  discrepancy.  [Ref. 
19: para.  2. 1.6. 7]  The  JCN  is  composed  of  four  parts: 

•  Organization  Code:  This  is  a  three-character 
alphanumeric  code  that  identifies  an  organization. 

•  Day:  This  is  the  three-character  part  of  the  Julian 
date  specifying  the  day  of  the  year.  This  is  the  date 
the  JCN  was  assigned  to  a  maintenance  action  and  does 
not  necessarily  reflect  the  date  on  which  work  was 
actually  started. 

•  Serial  Number:  The  serial  number  is  either  a  three 
character  number  that  runs  sequentially  from  001  to  999, 
or  a  three  character  alpha/numeric  number.  This  number 
is  normally  assigned  in  sequences  as  new  jobs  are 
initiated. 

•  Suffix:  The  JCN  suffix  is  a  structured  alpha/numeric 
code  added  to  the  basic  JCN  to  identify  a  sub-assembly 
or  sub-assembly  repair  action  performed  independently  of 
the  major  component  repair.  The  Suffix  is  used  only  for 
I-level  (Intermediate  level)  maintenance  functions 
regardless  of  where  the  maintenance  is  being  performed. 

2.  Work  Unit  Code  (WUC) 

The  WUC  is  a  one,  three,  five  or  seven  character  numeric 
or  alpha/numeric  code.  It  identifies  a  system,  subsystem. 
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set,  major  component,  repairable  siibassembly,  or  part  of  an 
end  item  being  worked  on.  The  system  code  is  the  first  two 
positions  of  the  WUC,  and  is  used  to  identify  the  system 
within  the  aircraft/equipment  on  which  work  is  being 
performed.  These  codes  are  listed  in  the  WUC  manual 
applicable  to  the  type,  model,  and  series  of  aircraft  being 
maintained.  [Ref  19: para.  2.1.6.12] 

3.  Transaction  Code  (TRANS) 

This  is  a  two-character  numeric  code  used  to  identify 
the  type  of  data  being  reported  [Ref  19: para.  9. 2. 4. 8]. 
Examples  of  commonly  used  transaction  codes  include: 

•  Transaction  Code  12:  On-Equipment  work,  including 
engines,  involving  non-repairable  components/ items 
documented  as  failed  parts. 

•  Transaction  Code  11:  On-Equipment  work  not  involving 
removal  of  defective  or  suspected  defective 
components/ items . 

•  Transaction  Code  23:  Removal  and  replacement  of  a 
defective  or  suspected  defective  repairable  component 
from  an  end  item. 

Appendix  P  of  reference  19  contains  a  complete  list  of  these 
codes  with  definitions. 

4.  Halfxinction  Description  Code  (NAL) 

This  is  a  three  character  numeric  code  used  to  describe 
the  malfunction  occurring  within  the  item,  assembly  or  sub- 
assembly  [Ref.  19: para.  9.2.4.12].  Examples  of  common 
malfunction  codes  include: 

•  782 :  Defective  or  damaged  tire  sidewall,  tread,  bead, 
etc. 
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•  381;  Leaking  -  internal  or  external. 

•  800;  No  defect  -  component  removed/reinstalled  to 

facilitate  other  maintenance. 

•  814 ;  Cannibalization  -  lack  of  replacement  material. 

A  complete  list  of  malfunction  codes  is  contained  in 
Appendix  I  of  reference  19.  [Ref.  19; para.  6. 2. 4. 6. 5] 

5.  Elapsed  Maintenance  Time  (EMT) 

EMT  is  defined  as  the  number  of  clock  hours  involved  in 
making  the  repair  (in  hours  and  tenths) .  Although  EMT  is 
directly  related  to  job  man-hours,  it  should  not  be  confused 
with  the  total  man-hours  required  to  complete  a  job,  for 
example,  if  four  persons  worked  together  for  2.5  hours  to 
make  a  repair,  the  total  maintenance  man-hours  expended 
would  be  10.0  and  the  EMT  would  be  2.5  hours.  [Ref . 19 ; para. 
9.2.4.15] 

6.  Data  Sample  Acquisition 

The  data  sample  provided  by  NALDA  was  comprised  of  the 
15  top  WUCs  for  the  F/A-18  as  determined  by  numbers  of 
maintenance  actions  processed  in  a  one  year  period.  All 
inspection  WUCs  were  mistakenly  excluded  from  the  data 
sample  by  members  of  the  NALDA  staff.  This  misunderstanding 
was  unknown  to  the  author  until  the  project  was  near 
completion.  The  statistical  effect  of  limiting  the  data 
sample  to  repair  actions  only,  instead  of  the  requested  top 
15  WUCs,  is  estimated  to  be  minor.  Data  elements  on  every 
organizational  level  F/A-18  maintenance  action  containing 


one  of  these  15  WUCs,  performed  during  the  twelve  month 
period  ending  in  August  of  1990,  were  included. 

The  data  sample  was  downloaded  from  the  NALDA  system, 
compressed,  and  loaded  onto  a  standard  1.2  MB,  5  1/4"  floppy 
disk  in  ASCII  format.  Upon  receipt,  the  data  was  un¬ 
compressed  using  the  PKUNZIP  program,  provided  on  the 
diskette,  and  loaded  onto  the  NPS  mainframe  computer. 

Readers  interested  in  obtaining  access  to  the  NALDA  system 
are  advised  to  contact  the  NALDA  Users  Assistance  Group  at 
Autovon  356-4454. 

7 .  Reasoning 

Based  Upon  the  author's  personal  experience  and  informal 
interviews  with  numerous  other  Aerospace  Maintenance  Duty 
Officers,  the  data  elements  listed  above  were  selected  for 
two  primary  reasons: 

•  The  most  critical  piece  of  information  required  for 
maintenance  scheduling  is  the  projected  EMT. 

•  For  comparison  purposes,  maintenance  actions  were  deemed 
to  be  identical  if  they  possessed  the  same  WUC,  MAL,  and 
TRANS  codes. 

EMT  was  selected  for  analysis  because  projected  EMT  is  a 
vital  piece  of  information  that  is  required  for  efficient 
maintenance  scheduling.  In  order  to  maximize  their 
activity's  aircraft  operational  readiness,  organizational 
level  maintenance  managers  seek  to  effectively  manage  their 
limited  manpower  and  material  resources.  The  challenge 
these  managers  face  several  times  a  day  is  to  wisely 
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schedule  repair  actions  so  as  to  maximize  the  number  of 
aircraft  available  to  meet  the  daily  flight  schedule. 
Projected  EMT  is  also  vital  to  the  effective  coordination 
and  management  of  shared  resources,  such  as  hangar  bay 
spots,  support  equipment  and  special  tools. 

Meeting  the  daily  flight  schedule  requirements  with 
safe,  flyable  and  properly  configured  aircraft  is  the  number 
one  priority  for  all  organizational  level  maintenance 
managers.  When  an  aircraft  returns  from  a  mission  with  one 
or  more  discrepancies,  the  maintenance  manager  must  quickly 
ascertain  the  nature  of  the  discrepancies  ("up"  or  "down")^ 
and  whether  or  not  his  activity  possesses  the  capability  bo 
repair  the  aircraft.  Assuming  that  the  aircraft  can  be 
repaired  by  the  activity,  the  maintenance  manager  must  then 
prioritize  the  discrepancies  and  determine  when  to  initiate 
the  repair  actions.  Aircraft  with  "down"  discrepancies  that 
can  be  quickly  repaired  (i.e.  a  short  projected  EMT)  and 
returned  to  an  "up"  status  are  generally  the  number  one  work 
priority,  so  as  to  provide  the  maximum  nuxtiber  of  aircraft 
assets  to  meet  the  daily  flight  schedule.  "Up"  discrep¬ 
ancies  and  "down"  aircraft  with  long  projected  EMT  will 
generally  receive  a  lower  work  priority,  often  being 
scheduled  for  repair  after  the  flight  schedule  has  been 


^Up  discrepancies  are  minor  problems  which  do  not  prevent  the 
aircraft  from  safely  performing  its  mission.  Down  discrepancies 
prevent  an  aircraft  from  flying  until  repaired. 
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completed.  A  more  detailed  description  of  the  maintenance 
control  decision  environment  is  contained  in  Reference  1. 

Several  rudimentary  statistical  analyses  were  performed 
upon  this  data  sample.  The  author  recognizes  that  the 
sample  may  not  be  representative  of  all  of  the  information 
contained  in  the  NALDA  databases  because  it:  (1)  is  limited 
to  one  aircraft  type,  (2)  is  composed  of  only  the  most 
common  maintenance  actions,  and  (3)  analysis  was  further 
limited  to  those  maintenance  actions  averaging  at  least  10 
occurrences  per  month.  Nevertheless,  the  analysis  did 
reveal  much  about  the  accuracy  and  consistency  of  NALDA 's 
FOJ  database . 

C.  NALDA  DATA  STRUCTURE/ FORMAT 

As  a  large  relational  database,  the  NALDA  system  can 
easily  retrieve  any  required  information  contained  in  any 
one  of  its  specific  databases.  While  the  computational 
capabilities  of  the  NALDA  system  are  limited  (means  and 
standard  deviations  can  be  calculated) ,  the  desired  data  can 
be  easily  downloaded  in  a  variety  of  formats  for  use  on 
other  more  powerful  computers. 

The  data  sample  used  in  preparing  this  thesis  was 
assembled  and  downloaded  onto  a  standard  floppy  disk  in 
ASCII  format.  The  NALDA  User  Assistance  Group  indicated  to 
the  author  their  willingness  to  provide  data  samples  to 
support  future  research.  Tasking  NAMO  to  regularly  organize 
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and  download  the  NALDA  data,  by  aircraft  type,  required  to 
develop  and  maintain  the  unique  databases  which  will  become 
essential  elements  of  the  ESAAMS  system  will,  in  the 
author's  opinion,  cause  only  a  minor  addition  to  their 
present  workload. 

Based  upon  this  author's  personal  experience  every 
expert  likely  to  be  interviewed  during  the  knowledge 
acquisition  phase,  particularly  senior  enlisted  maintenance 
managers,  should  be  thoroughly  familiar  with  the  data 
elements  contained  in  the  various  NALDA  databases.  Framing 
the  expert's  reasoning  in  terms  of  data  elements  contained 
in  the  various  NALDA  databases  should  prove  to  be  a  time 
consuming,  but  not  particularly  difficult,  task.  Some  items 
that  are  significant  factors  in  organizational  level 
aviation  maintenance  scheduling,  such  as  the  availability  of 
qualified  technicians  and  the  availability  of  required 
support  equipment  or  hangar  space,  are  not  contained  in  any 
of  the  NALDA  databases. 

D.  NALDA  DATABASE  MAINTENANCE 

Current  NALDA  instructions  call  for  the  various 
databases  to  be  updated  at  least  monthly  and  set  a  maximum 
data  time  lag  goal  of  60  days.  [Ref.  20]  During  interviews 
with  various  members  of  the  NALDA  Users  Assistance  Group  it 
was  stated  that  the  time  it  takes  the  data  concerning  a 
particular  maintenance  action  to  flow  through  the  three  MDS 
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cycles,  through  NAMSO,  and  eventually  in  to  the  various 
NALDA  databases  rarely  exceeds  90  days  and  currently 
averages  approximately  60  days.  [Ref.  21]  No  documentation 
was  available  to  support  this  statement,  however  these 
figures  of  60  and  90  days  corroborate  the  earlier  findings 
of  McCutcheon  [Ref.  22:p.  33]. 

It  is  important  to  note  that  the  majority  of  the 
maintenance  managers  interviewed  by  McCutcheon  felt  that 
reports  generated  from  data  that  was  60-90  days  old  were  of 
little  use  to  them  in  their  management  functions.  What  most 
maintenance  managers  desired  was  a  real-time  management 
information  system.  [Ref.  22 :p.  33]  This  author  knows  of 
no  such  real-time,  or  even  near  real-time,  system  which  will 
be  in  place  at  the  organizational  level  in  the  foreseeable 
future . 

E.  NALDA  DATA  ACCURACY 

With  the  understanding  that  the  accuracy  of  a  database 
is  directly  affected  by  the  guality  of  data  input,  this 
question  must  be  examined  from  two  aspects: 

•  how  accurately  is  the  information  transferred  from  the 
source  documents  and  entered  into  the  NALDA  databases? 

•  how  accurately  does  the  information  retrieved  from  the 
NALDA  databases  reflect  the  maintenance  actions  actually 
performed  in  the  fleet? 
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1.  How  Accurately  is  the  Information  Transferred  from 
the  Source  Documents  and  Entered  Into  the  NALDA 
Databases? 

Members  of  the  NALDA  Users  Assistance  Group  were 
confident,  during  interviews  with  the  author,  that  the  vast 
majority  of  keypunch  errors  were  caught  and  corrected  in  the 
local  cycle  of  the  MDS  system.  All  members  interviewed 
insisted  that  the  keypunch  error  rate  detected  at  NAMSO 
during  the  local/central  cycle  averaged  less  than  1%, 
although  this  figure  could  not  be  documented.  [Ref.  20]  The 
author,  based  on  personal  experience,  estimates  that  the 
claimed  1%  keypunch  error  rate  is  reasonably  accurate. 

2.  How  Accurately  does  the  Information  Retrieved  from 
the  Various  NALDA  Databases  Reflect  the  Maintenance 
Actions  Actually  Performed  by  the  Fleet  Activities? 

While  investigating  this  question  the  author  contacted  a 
representative  of  the  Aviation  Supply  Office  (ASO)  who  deals 
regularly  with  the  NALDA  databases.  This  representative 

voiced  significant  dissatisfaction  with  the  accuracy  of  the 

> 

NALDA  data.  ASO's  dissatisfaction  is  based  upon  three 
general  points  [Ref.  23]: 

•  Reports  generated  by  the  MDS  system  are  used  to  appraise 
unit  performance.  Therefore  it  is  in  the  originating 
units  interest  to  "fudge"  the  data,  particularly  data 
elements  affecting  unit  readiness. 


38 


•  Technicians  and  maintenance  managers  see  little  benefit 
from  the  reports  generated  by  the  MDS  system.  This 
leads  to  inaccurate,  inconsistent  and  incorrect 
reporting  of  maintenance  activities. 

•  Many  of  the  data  elements  within  the  MDS  system  are 
loosely  defined  and  interpreted,  leading  individual 
activities  document  identical  maintenance  actions 
differently. 

These  three  points  were  echoed  in  separate  interviews 
with  two  representatives  of  the  Naval  Air  Systems  Command. 
[Ref.  24  and  Ref.  25] 

Although  no  data  is  available  to  document  the  first 
point,  the  author,  based  upon  personal  experience,  will 
concede  that  "data  fudging"  does  occur.  The  majority  of 
"data  fudging"  occurs  with  Subsystem  Capability  Impact 
Reporting  (SCIR)  data  elements,  which  are  used  to  generate 
readiness  statistics.  The  extent  to  which  SCIR  data  will  be 
utilized  in  the  knowledge  acquisition  process  for  ESAAMS  is 
unknown  at  this  time.  SCIR  data  was  not  analyzed  as  part  of 
this  thesis. 

To  validate  the  other  two  points  a  large  number  of 
interview  summaries,  conducted  by  several  representatives  of 
the  Navy  Fleet  Material  Support  Office  (FMSO) ,  concerning 
the  MDS  system  and  NALDA  in  general,  and  NALDA’s  TDSA 
database  in  particular,  were  obtained.  This  series  of 
interviews  was  conducted  with  all  levels  of  the  chain  of 
command,  ranging  from  the  Systems  Command  level  down  to 
representatives  of  an  air  station  supply  department. 

Typical  of  the  comments  made  in  many  of  the  interviews  were 
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those  voiced  by  a  representative  of  the  Commander,  Naval  Air 
Forces  Atlantic  Fleet  who  viewed  the  NALDA  data  as 
inherently  inaccurate,  and  cited  three  primary  reasons  for 
this  inaccuracy  [Ref.  26]  : 

•  Part  Number  entries  are  unstructured:  Because  part 
numbers  are  assigned  by  the  manufacturer,  varying  widely 
in  length  and  combination  of  alpha  and  numeric 
characters,  no  easy  method  exists  to  screen  source 
documents  or  data  records  to  detect  incorrect  entries  or 
keypunch  errors. 

•  Code  changes:  Because  many  data  element  codes, 
particularly  Work  Unit  Codes,  are  often  updated  and 
changed,  many  data  entries  are  incorrectly  made  because 
the  technician  has  memorized  the  old  code  and  fails  to 
check  the  manual.  Additionally,  changing  data  element 
codes  vastly  complicate  the  task  of  analyzing  and 
comparing  historical  data. 

•  Loose  definitions!  Definitions  of  many  codes, 
particularly  malfunction  codes,  are  loose.  Individual 
intei^retations  of  which  data  code  is  appropriate  to  a 
particular  maintenance  action  will  have  a  significant 
impact  on  the  consistency  and  accuracy  of  the  databases. 

Another  problem  concerning  the  accuracy  of  the  NALDA 

databases  was  identified  by  a  representative  of  the  Naval 

Aviation  Depot  (NADEP)  at  Norfolk,  Virginia.  His  point  was 

that  documentation  of  repair  actions  of  individual 

components  was  inconsistent  at  most  NADEPs,  including  his 

own,  and  virtually  non-existent  if  the  item  was  repaired  by 

the  manufacturer  or  an  independent  contractor.  [Ref.  27] 

Because  this  project  is  not  likely  to  be  concerned  with  the 

repair  history  of  individual  components,  and  is 

concentrating  on  the  organizational  level,  this  problem 

should  have  little  impact  on  the  proposed  project. 
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During  another  interview  with  a  representative  of  the 
NADEP  Norfolk  A-6  engineering  staff,  the  results  of  an  audit 
of  NALDA's  TDSA  database  with  respect  to  a  specific 
airframes  modification  (AFC‘‘562)  revealed  that  of  1061 
entries,  130  (12%)  were  incorrect  [Ref.  28].  Discrepancies 
fell  into  two  general  categories: 

•  TDSA  showed  that  the  change  was  not  incorporated,  but 
the  change  actually  had  been  incorporated  on  the 
aircraft.  (60  discrepancies) 

•  TDSA  showed  that  the  change  was  incorporated,  but  the 
change  actually  had  not  been  incorporated  on  the 
aircraft.  (70  discrepancies) 

It  must  be  stressed  that  the  deficiencies  identified 
above  are  largely  opinions  based  on  the  personal  experiences 
of  the  interview  subjects,  and  are  not  conclusive  proof  that 
the  data  contained  in  the  NALDA  system  is  inaccurate.  It 
should  also  be  noted  that  any  deficiencies  that  may  exist  in 
the  NALDA  databases  appear  to  be  largely  the  result  of  "user 
error"  throughout  the  fleet  and  mismanagement  of  the  MDS 
program,  and  are  not  the  fault  or  specific  responsibility  of 
the  NALDA  managers.  Further  evidence  concerning  the 
accuracy  of  the  NALDA  databases  will  be  discussed  in  later 
sections  of  this  chapter. 

F.  THE  AFFECT  OF  GEOGRAPHICAL  LOCATION  OF  ORIGINATING 
ACTIVITIES  ON  EMT 

To  ascertain  the  effect  of  the  geographic  location  of 
the  originating  activities  on  EMT,  the  data  records  from 
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the  sample  provided  by  NAMO  were  separated,  using  the 
organization  codes  located  within  the  job  control  number 
(JCN) ,  into  East  coast  and  West  cost  maintenance  actions. 
The  mean  EMT  for  92  individual  maintenance  actions  for  both 
coasts  were  compared  and  the  variances  analyzed.  The 
following  definitions  were  established: 

^0  •  ^^easc“^^ll^est 

Ho  is  the  hypothesis  we  wished  to  test  and  is  referred  to  as 
the  null  hypothesis.  The  rejection  of  Ho  leads  to 
acceptance  of  the  alternative  hypothesis,  denoted  Hj.  An 
example  listing  of  the  ANOVA  results  is  contained  in  Figure 
2.2 

Of  the  92  maintenance  actions  analyzed,  42  (47%) 
indicated  a  probability  of  less  than  5%  that  the  two  means 
were  equal.  Assuming  that  this  is  a  representative  sample, 
the  results  indicate  that  the  geographical  location  of  the 
originating  activity  does  significantly  affect  the  NALDA 
databases.  If  we  are  to  use  historical  EMT  data  to  project 
EMT,  the  expert  system  will  need  to  take  this  fact  into 


^Complete  listings  of  these  ANOVA  results  will  be  maintained 
on  file  by  the  author  and  Professor  Martin  J.  McCaffrey  of  the 
Naval  Postgraduate  School,  Monterey  California. 
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consideration.  NALDA  EMT  data  will  most  likely  be  divided 
into  East  and  West  coast  databases. 

A  closer  examination  of  the  complete  ANOVA  results 
listings  and  the  example  contained  in  Figure  2  reveals  two 
other  interesting  factors.  First,  despite  a  low  statistical 
probability  that  the  true  East  and  West  coast  means  were 
equal,  many  of  the  sample  means  differed  by  0.25  hours  or 
less.  While  a  variation  of  0.25  hours  may  translate  into 
significant  statistical  difference  between  means,  from  a 
practical  maintenance  manager's  viewpoint  this  difference  is 
essentially  insignificant  for  many  maintenance  actions. 

The  second  factor  noted  was  the  wide  variation  in  the 
frequency  of  occurrence  of  identical  maintenance  actions 
between  the  East  and  West  coasts.  With  both  coasts 
containing  roughly  the  same  number  of  F/A-18  aircraft,  the 
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niunber  of  identical  maintenance  actions  performed  on  each 
coast  was  expected  to  be  approximately  equal.  While  the 
majority  of  maintenance  actions  analyzed  displayed  a  rough 
parity  in  terms  of  number  of  maintenance  actions  performed 
on  both  coasts,  ratios  of  two-to-one  and  three-to-one  were 
common,  with  the  highest  ratio  noted  exceeding  twenty- to- 
one.  The  example  results  displayed  in  Figure  1  show  a  near 
three-to-one  ratio  between  the  west  and  east  coasts. 

Neither  the  East  or  West  coast  consistently  dominated  its 
counterpart  when  the  frequencies  were  imbalanced.  This 
inconsistency  lends  support  to  the  allegations  listed  above 
that  the  definitions  of  some  MDS  code  elements  are  loose. 

If  activities  performing  identical  maintenance  actions 
continue  to  document  them  differently  because  of  loose  code 
definitions,  inconsistencies  and  inaccuracies  in  the 
database  will  persist. 

G.  UPDATE  FREQUENCY  OF  AN  EXPERT  SYSTEM  DATABASE,  FOUNDED 

UPON  THE  NALDA  DATABASES 

To  evaluate  how  often  the  knowledge  base  of  the  ESAAHS 
system  will  require  updates  from  the  NALDA  databases,  the 
maintenance  actions  documented  in  the  data  sample  were 
grouped  by  month,  using  the  Julian  date  component  of  the 
JCN.  Mean  EMTs  for  each  month  for  each  specific  maintenance 
action  were  compared  and  the  variances  analyzed.  Months 
were  numbered  sequentially,  one  through  twelve,  and  the  mean 


EMT  for  each  month  was  calculated.  The  null  hypothesis.  Ho, 
was  defined  as  follows: 


•  -=1*12 


Hi,  the  alternative  hypothesis,  was  defined  as  at  least  two 
of  the  means  not  being  equal.  An  example  listing  of  the 
ANOVA  results  is  contained  in  Figure  2.®  Of  105  separate 
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maintenance  actions  examined  ,  32  (30%)  indicated  a 
probability  of  less  than  5%  that  the  true  means  of  all 
twelve  months  were  equal.  The  ANOVA  results  indicate  that 


^Complete  listings  of  these  ANOVA  results  will  be  maintained 
on  file  by  the  author  and  Professor  Martin  J.  McCaffrey  of  the 
Naval  Postgraduate  School,  Monterey  California. 
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updates  of  the  knowledge  base  for  at  least  these  32 
maintenance  actions  will  need  to  be  conducted  more 
frequently  than  once  per  year.  The  actual  determination  of 
how  often  to  update  the  databases  used  to  develop  and 
maintain  the  knowledge  base  for  the  proposed  expert  system 
will  require  a  more  detailed  statistical  and  cost/benefit 
analysis  and  is  beyond  the  scope  of  this  report. 

It  was  noted  that  while  small  variations  (less  than  .25 
hours)  between  the  monthly  mean  EMTs  and  the  mean  EHT  for 
the  twelve  month  period  may  have  been  statistically 
significant  for  several  of  the  maintenance  actions  examined, 
they  were  again,  from  a  practical  user's  viewpoint, 
essentially  insignificant  for  many  maintenance  actions. 


H.  SUMHARY 

This  chapter  has  presented  the  reader  with  a  large 
amount  of  information  concerning  the  accuracy  and 
consistency  of  the  various  NALDA  databases  and  their  ability 
to  support  the  unique  databases  which  will  become  essential 
elements  of  the  ESAAMS  system.  The  key  points  to  take  from 
this  chapter  are: 

•  Because  every  expert  in  Naval  Aviation  maintenance 
management  is  thoroughly  familiar  with  the  MDS  system,  a 
database,  such  as  those  maintained  by  NALDA,  can  easily 
break-out  the  data  required  to  validate  the  expert's 
reasoning. 

•  The  data  contained  in  the  various  NALDA  databases  is 
updated  at  least  monthly,  with  an  average  time  lag  of  60 
days  between  completion  of  a  particular  maintenance 
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action  and  inclusion  of  the  data  in  the  NALDA  system. 
While  real-time,  or  near  real-time,  database  updates  may 
be  desirable,  such  a  system  may  not  be  cost  effective 
and  is  not  foreseen  to  appear  at  the  organizational 
level  in  the  near  future. 

•  While  the  accuracy  of  the  transfer  of  data  from  source 
documents,  through  the  three  MDS  cycles,  and  into  the 
NALDA  databases  appears  to  be  very  good,  the  ability  of 
the  MDS  data  elements  to  accurately  reflect  the  actual 
maintenance  actions  being  performed  in  the  fleet  is  in 
doubt.  This  doubt  is  primarily  caused  by  the  loose 
definitions  of  some  of  the  data  elements,  leading  to 
inconsistent  documentation  of  identical  maintenance 
actions . 

•  Geographical  location  of  originating  activities,  even 
when  generalized  to  just  the  east  and  west  coasts,  has  a 
substantial  affect  upon  the  data  elements  most  likely  to 
affect  aircraft  maintenance  scheduling. 

•  While  a  large  number  of  maintenance  actions  sampled 
displayed  adequate  consistency  of  mean  EMT  figures 
throughout  a  twelve  month  period,  30%  did  not.  At  least 
this  30%,  and  perhaps  more,  will  require  database 
updates  more  often  than  once  per  year. 
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IV.  CONCLUSIONS 


A.  SUMMARY 

In  chapter  II  we  learned  that  the  task  of  acquiring  the 
knowledge  required  to  construct  a  useful  and  practical 
expert  system  is  the  most  difficult  step  in  the  entire 
expert  system  development  process.  While  a  number  of 
knowledge  acquisition  techniques  are  available,  the  growing 
size  of  knowledge  bases  is  necessitating  a  move  to  automate 
some  portion  of  the  knowledge  acqpaisition  process. 

A  number  of  expert  system  shells  currently  available  on 
the  commercial  market  offer  automated  knowledge  acquisition 
modules.  These  modules  possess  the  capability  to 
automatically  generate  knowledge  base  rules  based  upon  the 
data  contained  in  various  database  files.  The  key  to  any 
automated  knowledge  acquisition  effort  is  the  accuracy  of 
the  data  contained  in  the  databases  and  the  connectability 
of  the  expert  system  shell  used  to  develop  the  system. 

In  Chapters  III  and  IV  we  examined  the  NALDA  databases 
in  an  effort  to  determine  their  suitability  for  use  as  the 
foundation  for  the  knowledge  base  of  our  proposed  expert 
system.  Several  weaknesses  in  the  source  of  NALDA 's  data. 
Naval  Aviation's  MDS  system,  were  identified.  Mean  EMT  was 
assumed  to  be  the  most  critical  data  element  for  aircraft 
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maintenance  discrepancy  scheduling  maintained  by  NALDA. 
Rudimentary  statistical  analysis  revealed  that  mean  ENT:  (1) 
is  significantly  affected  by  geographical  location,  and  (2) 
for  a  notable  percentage  of  maintenance  actions,  varies 
significantly  during  a  twelve  month  period. 

Despite  the  drawbacks  identified  above,  the  NALDA 
databases  are  unic[uely  qualified  to  serve  as  the  foundation 
for  the  knowledge  base  of  our  proposed  expert  system.  While 
the  inconsistencies  identified  among  the  data  may  inhibit  or 
preclude  the  automation  of  the  knowledge  acquisition 
process,  it  does  not  make  the  databases  unusable.  Many  of 
the  inconsistencies,  while  statistically  significant,  are 
insignificant  from  a  practical  users  point  of  view. 

NALDA  is  uniquely  qualified  to  provide  the  information 
required  to  serve  as  the  foundation  for  the  EMT  database  of 
our  proposed  expert  system  for  the  following  reasons: 

•  As  Naval  Aviation's  central  repository  of  logistical  and 
maintenance  data,  NALDA  is  the  only  conceivable  source 
for  much  of  the  data  required. 

•  Every  aircraft  maintenance  expert  likely  to  be 
interviewed  during  the  knowledge  acquisition  process 
will  be  thoroughly  familiar  with  the  data  elements 
contained  in  the  various  NALDA  databases.  These  data 
elements  can  thus  serve  as  a  "common  language"  when 
expert  reasoning  are  consolidated. 

•  The  source  of  much  of  NALDA 's  data,  the  three  MDS 
cycles,  are  in  place  and  functioning  throughout  the  U.S. 
Navy.  Despite  any  shortcomings  the  system  may  possess, 
replacing  it  or  duplicating  it  would  be  prohibitively 
expensive. 
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•  The  NALDA  system  is  organized  to  respond  to  ad  hoc  data 
inquiries.  Any  data  required  during  knowledge 
acquisition  can  be  quickly  retrieved  from  one  or  more  of 
the  various  databases,  and  downloaded  in  a  variety  of 
communication  formats. 


B.  RECOMMENDATIONS 

Based  upon  the  previous  discussion,  it  is  svibmitted  that 
utilizing  NALDA  as  the  foundation  of  a  knowledge  base  of  an 
expert  system  to  be  used  for  aircraft  maintenance 
discrepancy  scheduling  is  feasible.  The  improved  management 
effectiveness,  and  potential  for  improved  aircraft 
operational  availability,  that  such  an  expert  system  offers 
warrant  continued  research  and  development  efforts. 

Areas  that  warrant  further  research  include: 

•  Easy  access  to  NALDA  terminals  would  allow  maintenance 
managers  to  design  reports  to  provide  them  with  the 
information  they  feel  they  need,  and  provide  them  with 
quicker  feedback.  A  feasibility  study  examining  the 
costs  and  potential  benefits  available  from  placing 
remote  NALDA  access  terminals  in  every  organizational 
level  maintenance  activity  should  be  initiated. 

•  A  comprehensive  examination  of  the  factors  critical  to 
the  success  of  maintenance  managers  should  be  conducted. 
This  examination  should  seek  to  identify  the  specific 
factors,  data  elements,  and  information  required  for  the 
knowledge  base  of  the  proposed  expert  system. 

•  A  prototype  expert  system  should  be  constructed  to 
demonstrate  the  feasibility  and  benefits  available  from 
the  use  of  such  a  system  by  maintenance  managers.  The 
break-out  of  NALDA  data  on  a  specific  aircraft  type,  and 
the  consolidation  of  this  data  into  a  separate  database 
to  be  used  for  the  ESAAMS  prototype,  is  required. 
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APPENDIX  A. 

A.  DATA  SAMPLE 


NAM*S2(02S 

MS9300 

11 

000 

.9 

MASa9S100i7 

9039300 

000 

3.4 

1.7 

MAM9S1MSI 

M39300 

10 

014 

1.0 

MAStfSWOSl 

MS9500 

11 

000 

1.2 

HAM«S2(022 

5039300 

11 

000 

4.0 

2.0 

AE(9009712t 

5039300 

11 

Oil 

1.2 

•  W 

AE(900»7129 

5039300 

11 

oil 

1.2 

•  V 

AE(900f71S0 

5039300 

11 

oil 

1.2 

•  w 

.AE4900971S1 

5039300 

11 

ail 

1.2 

•  V 

AE«)00)71S2 

5039300 

11 

oil 

1.2 

•  ^ 

AE<9010«St0 

5039300 

11 

ail 

1.0 

«  V 

AEi90t0«ail 

5039300 

11 

an 

1.0 

•  V 

AEi9010i«12 

5039300 

11 

on 

1.5 

AEilOtOMlS 

5039300 

11 

on 

1.5 

•  V 

AE490097127 

5039300 

11 

on 

1.2 

•  ^ 

AEitOlOiSlS 

5039300 

11 

on 

1.5 

•  V 

AEi90te702S 

5039300 

11 

on 

1.9 

1.0 

AEifOlOMlA 

5039300 

11 

on 

1.5 

•  V 

AEifOOnOU 

5039300 

11 

004 

.5 

»  9 

AES901202<4 

5039300 

11 

000 

.4 

•  ^ 

AES90n40(4 

5039300 

11 

000 

.4 

»  % 

AE590I 14102 

5039300 

11 

000 

.4 

•  # 

AESOOlUlOa 

5039300 

23 

290 

1.1 

1.1 

AE59011420S 

5039300 

11 

000 

.4 

0% 

AE590120245 

5039300 

11 

000 

.4  • 

*  “ 

AE(90107029 

5039300 

11 

on 

1.0 

AES901208St 

5039300 

10 

014 

1.0 

1.0 

AE«900940ir 

5039300 

23 

374 

1.2 

•  w 

AEI9009401S 

5039300 

11 

004 

.5 

•  # 

AE«9009401( 

5039300 

10 

014 

.5 

»  9 

AE490094017 

5039300 

11 

004 

.5 

AE(900950S5 

5039300 

11 

004 

.5 

AE(9009S03( 

5039300 

11 

004 

.5 

AE(901I2091 

5039300 

11 

014 

.4 

.2 

AEt901 12091 

5039300 

10 

014 

3.0 

1.5 

AE490114049 

5039300 

11 

on 

1.2 

AE490114070 

5039300 

11 

on 

1.2 

AE4901 14071 

5039300 

11 

on 

1.2 

AE(90114072 

5039300 

11 

on 

1.2 

AE(90114074 

5039300 

11 

on 

1.2 

AE49011407S 

5039300 

11 

on 

1.2 

AE»9011407t 

5039300 

11 

on 

1.2 

AE(90115004 

5039300 

11 

on 

1.2 

AE(90115010 

5039300 

11 

on 

1.2 

AE(9011S011 

5039300 

11 

on 

1.2 

54 


INITIAL  DISTRIBUTION  LIST 


No.  Copies 

1.  Defence  Technical  Information  Center  2 

Cameron  Station 

Alexandria,  VA  22304-6145 

2.  Library,  Code  52  2 

Naval  Postgraduate  School 

Monterey,  CA  93943-5002 

3.  Defense  Logistics  Studies  Information  Exchange  1 

United  States  Army  Logistics  Management  Center 

Fort  Lee,  VA  23801-6043 

4.  LCDR  John  D.  Burpo,  USN 
103  South  Main  St. 

Ringwood,  OK  73768 

5.  Professor  M.  J.  McCaffrey,  (Code  AS/MF)  5 

Department  of  Administrative  Sciences 

Naval  Postgraduate  School 
Monterey,  CA  93943 

6.  Professor  T.  X.  Bui,  (Code  AS/BD)  1 

Department  of  Administrative  Sciences 

Naval  Postgraduate  School 
Monterey,  CA  93943 

7.  Commanding  Officer  1 

Naval  Aviation  Maintenance  Office  (Code  622C) 

Naval  Air  Station 
Patuxent  River,  MD  20670 

8.  Commanding  Officer  1 

Aviation  Supply  Office  (Code  045401) 

700  Robbins  Avenue 
Philadelphia,  PA  19111 

9.  Commanding  Officer  1 

Naval  Air  Systems  Command  (Code  PMA-205-33) 

Naval  Air  Systems  Command  Headquarters 
Washington,  DC  20361 


55 


